Papers with NLP problems
Dive into Deep Learning for Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | GluonNLP is a powerful new toolkit that automates the most laborious aspects of deep learning for NLP. |
| Approach: | This hands-on tutorial demonstrates how to scale unsupervised pre-training techniques with Apache MXNet and GluonNLP. |
| Outcome: | This hands-on tutorial examines the challenges of scaling these models and algorithms effectively with Apache MXNet and GluonNLP. |
Deep Adversarial Learning for NLP (N19-5)
Copied to clipboard
| Challenge: | Adversarial learning is a game-theoretic learning paradigm that has achieved huge successes in the field of Computer Vision recently. |
| Approach: | This tutorial introduces the foundations of deep adversarial learning and some practical problems and solutions in NLP. |
| Outcome: | This tutorial introduces the foundations of deep adversarial learning and some practical problems and solutions in NLP. |
Meta Learning and Its Applications to Natural Language Processing (2021.acl-tutorials)
Copied to clipboard
| Challenge: | Meta-learning is a new technique that aims to learn better learning algorithms, including better parameter initialization, optimization strategy, network architecture, distance metrics, and beyond. |
| Approach: | This tutorial introduces Meta-learning approaches and the theory behind them, and then reviews the works of applying this technology to NLP problems. |
| Outcome: | This tutorial will introduce Meta-learning approaches and the theory behind them, and then review the works of applying this technology to NLP problems. |
Deep Learning on Graphs for Natural Language Processing (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | Graph Neural Networks (GNNs) are powerful tools for non-Euclidean data modeling and are used in many graph-related NLP tasks. |
| Approach: | This tutorial will cover applying deep learning on graph techniques to NLP using Graph Neural Networks (GNNs) Graph4NLP is the first library for researchers and practitioners for easy use of GNNs for various NLP tasks. |
| Outcome: | This tutorial will cover the latest developments in deep learning on graph techniques and their applications in various NLP tasks. |
Deep Latent Variable Models of Natural Language (D18-3)
Copied to clipboard
| Challenge: | In this tutorial, we will discuss the challenges of applying neural variational inference to NLP problems. |
| Approach: | The tutorial will cover deep latent variable models in the case where exact inference over the latent variables is tractable. |
| Outcome: | The proposed tutorial will cover deep latent variable models in the case where inference cannot be performed tractably and when it is not . |
CausalNLP Tutorial: An Introduction to Causality for Natural Language Processing (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | Establishing causal relationships is a fundamental goal of scientific research . lack of clear definitions, notations, benchmark datasets, and challenges remains . |
| Approach: | They introduce the fundamentals of causal discovery and causal effect estimation to the natural language processing audience and provide an overview of causal perspectives to NLP problems. |
| Outcome: | This tutorial introduces the fundamentals of causal discovery and causal effect estimation to the natural language processing audience and provides an overview of causal perspectives to NLP problems. |
Multi-Task Learning for Argumentation Mining in Low-Resource Settings (N18-2)
Copied to clipboard
| Challenge: | Argument component identification is difficult for trained annotators to perform in a new domain or to develop new AM tasks. |
| Approach: | They investigate whether multi-task learning can improve performance on AM problems . they found that MTL performs particularly well when little training data is available for the main task . |
| Outcome: | The proposed approach performs better when little training data is available for the main task, a common scenario in AM. |
Learning Compressed Sentence Representations for On-Device Text Processing (P19-1)
Copied to clipboard
Dinghan Shen, Pengyu Cheng, Dhanasekar Sundararaman, Xinyuan Zhang, Qian Yang, Meng Tang, Asli Celikyilmaz, Lawrence Carin
| Challenge: | Existing methods for learning sentence embeddings assume they are continuous and real-valued. |
| Approach: | They propose four different strategies to transform continuous and generic sentence embeddings into a binarized form while preserving their rich semantic information. |
| Outcome: | The proposed methods reduce storage requirements by over 98% and improve performance on downstream tasks. |
Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge Extraction from Spine MRI Reports (2022.naacl-industry)
Copied to clipboard
Arijit Sehanobish, McCullen Sandora, Nabila Abraham, Jayashri Pawar, Danielle Torres, Anasuya Das, Murray Becker, Richard Herzog, Benjamin Odry, Ron Vianu
| Challenge: | Pretrained Transformer based models finetuned on domain specific corpora have changed the landscape of NLP but training or fine-tuning these models for individual tasks can be time consuming and resource intensive. |
| Approach: | They propose to use pretrained Transformer based models finetuned on domain specific corpora to train models for individual tasks. |
| Outcome: | The proposed model can match the performance of a task specific model when the task specific models show similar representations across all of their hidden layers and their gradients are aligned, i.e. their gradient follows the same direction. |
Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Tasks (2023.acl-srw)
Copied to clipboard
| Challenge: | Current approaches to OOD detection in NLP are not yet sufficiently sensitive to capture all samples characterized by various types of distributional shifts. |
| Approach: | They evaluated eight methods that are easily integrable into existing NLP systems and require no additional OOD data or model modifications. |
| Outcome: | The proposed methods are easily integrable into existing NLP systems and require no additional OOD data or model modifications. |
Extractive NarrativeQA with Heuristic Pre-Training (D19-58)
Copied to clipboard
| Challenge: | Automated question answering (QA) from text remains a challenge for humans . a striking gap exists between machine and human performance on NLP tasks . |
| Approach: | They propose a heuristic extractive version of a data set to solve the problem of answer extraction rather than generation. |
| Outcome: | The proposed model outperforms previous models on summary-level QA from full narratives and on the METEOR metric. |
ferret: a Framework for Benchmarking Explainers on Transformers (2023.eacl-demo)
Copied to clipboard
| Challenge: | Existing methods for interpreting transformer outputs are scattered and hard to operationalize. |
| Approach: | They propose a Python library to simplify the use and comparisons of XAI methods on transformers. |
| Outcome: | The proposed method provides better explanations and is preferable in the context of transformer models. |
Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement (P18-1)
Copied to clipboard
| Challenge: | a number of post hoc explanation methods for deep neural networks have been proposed . due to the complexity of the DNNs they explain, these methods are necessarily approximations and come with their own sources of error. |
| Approach: | They propose two evaluation paradigms that cover two important classes of NLP problems . they propose LIMSSE, LRP and DeepLIFT as the most effective explanation methods . |
| Outcome: | The proposed methods are most effective for explaining deep neural networks in NLP . the proposed methods can explain complex models without manual annotation . |
Saturated Transformers are Constant-Depth Threshold Circuits (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work shows that transformers with hard attention are limited in power, but hard attention is a strong assumption. |
| Approach: | They propose a generalization of hard attention that captures attention patterns in transformers with saturated attention. |
| Outcome: | The proposed generalization of hard attention more closely captures the attention patterns learnable in practical transformers. |
Meta Learning for Natural Language Processing: A Survey (2022.naacl-main)
Copied to clipboard
| Challenge: | Meta-learning is an emerging field in machine learning, but there is no systematic survey of these approaches in NLP. |
| Approach: | They propose to introduce meta-learning and the common approaches and summarize their work and review their work in the NLP community. |
| Outcome: | The proposed methods improve performance in many NLP tasks but are limited to domains, languages, countries, or styles. |
HappyDB: A Corpus of 100,000 Crowdsourced Happy Moments (L18-1)
Copied to clipboard
Akari Asai, Sara Evensen, Behzad Golshan, Alon Halevy, Vivian Li, Andrei Lopatenko, Daniela Stepanov, Yoshihiko Suhara, Wang-Chiew Tan, Yinzhan Xu
| Challenge: | Recent research has focused on developing technologies that help users incorporate the findings of the science of happiness into their daily lives. |
| Approach: | They crowd-sourced HappyDB, a corpus of 100,000 happy moments, and applied several state-of-the-art analysis techniques to analyze HappyDB. |
| Outcome: | The proposed technology can understand how people express their happy moments in text and analyze them using state-of-the-art techniques. |
Incorporating Contextual and Syntactic Structures Improves Semantic Similarity Modeling (D19-1)
Copied to clipboard
| Challenge: | Semantic similarity modeling is central to many NLP problems such as question answering. |
| Approach: | They propose a pairwise word interaction model with syntactic structure priors to explore their effectiveness. |
| Outcome: | Extensive evaluations on eight benchmark datasets show that incorporating structural information improves over strong baselines. |
Learning What to Share: Leaky Multi-Task Network for Text Classification (C18-1)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning suffer from the interference between tasks because they lack selection mechanism for feature sharing. |
| Approach: | They propose a multi-task convolutional neural network with the Leaky Unit which has memory and forgetting mechanism to filter the feature flows between tasks. |
| Outcome: | The proposed model can filter feature flows between tasks and improve performance on five datasets. |
Can Unconfident LLM Annotations Be Used for Confident Conclusions? (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown high agreement with human raters across a variety of tasks, demonstrating potential to ease the challenges of human data collection. |
| Approach: | They propose a method that combines LLM annotations and LLM confidence indicators to strategically select which human annotations to use. |
| Outcome: | The proposed method produces accurate estimates and valid confidence intervals while reducing the number of human annotations by over 25%. |
Deep Contextualized Word Representations (N18-1)
Copied to clipboard
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer
| Challenge: | a new type of deep contextualized word representation is proposed for language understanding problems . word vectors are learned functions of the internal states of a deep bidirectional language model . |
| Approach: | They propose a new type of deep contextualized word representation that models complex features of word use and how they vary across linguistic contexts. |
| Outcome: | The proposed representations improve the state of the art across six challenging NLP problems. |
On Robustness of Neural Semantic Parsers (2021.eacl-main)
Copied to clipboard
| Challenge: | Semantic parsing maps natural language (NL) utterances into logical forms (LFs) adversarial examples are created by adding tiny perturbations to inputs but can severely deteriorate model performance. |
| Approach: | They propose to construct robustness test sets based on existing benchmark corpora and to evaluate the effect of data augmentation. |
| Outcome: | The proposed method measures the performance of the proposed parsers on robustness test sets and evaluates the effect of data augmentation. |
Phrase Retrieval Learns Passage Retrieval, Too (2021.emnlp-main)
Copied to clipboard
| Challenge: | Dense retrieval methods have shown great promise over sparse methods in a range of NLP problems. |
| Approach: | They propose to use dense phrase retrieval to learn coarse-level retrieval including passages . they show phrase retrievals can be fine-tuned for more coarse-grained retrieval units . |
| Outcome: | The proposed method improves passage retrieval accuracy and QA performance with fewer passages. |
A Survey of Active Learning for Natural Language Processing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature surveys on active learning for NLP are too specific or too general, covering deep active learning. |
| Approach: | They propose to use active learning to improve model learning and annotation cost for NLP problems. |
| Outcome: | The proposed approach is based on a large dataset of data-driven machine learning models. |
How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work probing pre-trained language models for downstream tasks is difficult to explain . a growing body of research is devoted to understanding what linguistic properties these language models have acquired. |
| Approach: | They propose a procedure and analysis method that takes a hypothesis of how a transformer-based model might encode a linguistic phenomenon and tests its validity. |
| Outcome: | The proposed method tests a hypothesis that some attention heads will consistently attend from a word in negation scope to the negation cue. |
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on sentence-level inference, which limits its application in downstream NLP problems. |
| Approach: | They propose to construct a large-scale dataset for document-level NLI that can be used to study NLP problems. |
| Outcome: | The proposed model performs well on popular sentence-level benchmarks and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity. |
bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark (2023.acl-long)
Copied to clipboard
Momchil Hardalov, Pepa Atanasova, Todor Mihaylov, Galia Angelova, Kiril Simov, Petya Osenova, Veselin Stoyanov, Ivan Koychev, Preslav Nakov, Dragomir Radev
| Challenge: | bgGLUE is a benchmark for evaluating language models on natural language understanding (NLU) tasks in Bulgarian. |
| Approach: | They propose to use a benchmark to evaluate language models on NLU tasks in Bulgarian. |
| Outcome: | The proposed model performs well on sequence labeling tasks, but there is room for improvement for tasks that require more complex reasoning. |
Auto-hMDS: Automatic Construction of a Large Heterogeneous Multilingual Multi-Document Summarization Corpus (L18-1)
Copied to clipboard
| Challenge: | Existing datasets for automatic text summarization are small and focused on newswires. |
| Approach: | They propose to automatically generate a large multilingual multi-document summarization corpus using Wikipedia articles as summaries and to automatically search for appropriate source documents. |
| Outcome: | The proposed corpus contains 7,316 topics in English and German with different summary lengths and number of source documents. |
Bridging the Gap between Relevance Matching and Semantic Matching for Short Text Similarity Modeling (D19-1)
Copied to clipboard
| Challenge: | Existing techniques for relevance and semantic matching cannot be easily adapted to the other. |
| Approach: | They propose a model that incorporates a hybrid encoder module, a relevance matching module and co-attention mechanisms that capture context-aware semantic relatedness. |
| Outcome: | The proposed model incorporates a hybrid encoder module, a relevance matching module and co-attention mechanisms that capture context-aware semantic relatedness. |
NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data (2024.emnlp-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a core component of natural language processing, present in a variety of applications such as medical coding, financial news analysis, or legal documents parsing. |
| Approach: | They propose to use Large Language Models (LLMs) to create NuNER, a compact language representation model specialized in the Named Entity Recognition task. |
| Outcome: | The proposed model outperforms similar-sized foundation models in the few-shot regime and is based on a human-annotated dataset. |
Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start (2020.emnlp-main)
Copied to clipboard
| Challenge: | a current approach to solving NLP problems is to build a problem-specific dataset . current approaches do not allow for transforming tasks into textual entailment . |
| Approach: | They propose a pretrained textual entailment system that can generalize across domains . they argue that when is it worth transforming an NLP task into textual detailment? |
| Outcome: | The proposed model can generalize across domains with few examples, the authors argue . they show that it can be used for several downstream NLP tasks with limited annotations . |
A Multitask Learning Approach for Diacritic Restoration (2020.acl-main)
Copied to clipboard
| Challenge: | Diacritics are used to specify pronunciations and meanings in many languages like Arabic. |
| Approach: | They propose to use multi-task learning to optimize diacritic restoration with related NLP problems . they use Arabic as a case study since it has sufficient data resources for tasks . |
| Outcome: | The proposed model outperforms baseline models and is comparable to the state-of-the-art models. |
Treepiece: Faster Semantic Parsing via Tree Tokenization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Autoregressive (AR) encoder-decoder neural networks are slow in sequential prediction of natural language to machine-readable parse trees. |
| Approach: | They propose a technique that tokenizes a parse tree into subtrees and generates one subtrea per decoding step. |
| Outcome: | The proposed approach shows 4.6 times faster decoding speed and comparable speed but significantly higher accuracy compared to non-autoregressive (NAR) models. |
PALS: Personalized Active Learning for Subjective Tasks in NLP (2023.emnlp-main)
Copied to clipboard
Kamil Kanclerz, Konrad Karanowski, Julita Bielaniewicz, Marcin Gruza, Piotr Miłkowski, Jan Kocon, Przemyslaw Kazienko
| Challenge: | Personalized active learning techniques can be used to learn subjective NLP problems . to acquire training data, texts are often randomly assigned to users for annotation . |
| Approach: | They propose to apply an active learning paradigm to a personalized context to learn preferences . they validated their techniques on a Wiki discussion text labeled with aggression and toxicity . |
| Outcome: | The proposed methods outperform random selection and random selection by 30% on three datasets. |
Softmax Tree: An Accurate, Fast Classifier When the Number of Classes Is Large (2021.emnlp-main)
Copied to clipboard
| Challenge: | Classification problems with thousands or more classes occur in NLP, for example language models or document classification. |
| Approach: | a new algorithm uses a binary tree with sparse hyperplanes and small softmax classifiers at the leaves to predict the top class. |
| Outcome: | The proposed model is faster at inference because the input follows a single path to a leaf and the softmax classifier operates on a small subset of the classes. |
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2023.acl-long)
Copied to clipboard
Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang, Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang, Jason Phang, Samuel R. Bowman
| Challenge: | Getting sociological beliefs wrong can slow research and lead to wasted effort, missed opportunities, and needless fights. |
| Approach: | They present the results of the NLP Community Metasurvey, run from May to June 2022. |
| Outcome: | The NLP community metasurvey elicited opinions on controversial issues from May to June 2022. |
Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriver + Flan-T5 are popular retriever-augmented language models for a variety of tasks. |
| Approach: | They evaluate the strengths and weaknesses of kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriver + Flan-T5 in reasoning over retrieved statements across different tasks. |
| Outcome: | The proposed models do not exhibit strong reasoning even when provided with only the required statements. |